Optical correlator assisted detection of calcifications for breast biopsy

ABSTRACT

Ultrasonographic imaging of small calcifications or similar hard bodies distributed in human breast tissue is enhanced by correlating an ultrasonographic data set with a radiographic image of the same region of interest. A “constellation” or cluster of small calcifications is distinguished from speckle noise by the cross-correlation, which is quite sensitive to the coincidence of a pattern of distributed small targets in both the ultrasonographic and radiographic images, notwithstanding the presence of random noise. An optical correlator is preferably used to perform high speed cross-correlations. The three-dimensional position of an individual calcification is preferably found by projecting from an identified point in the radiographic image, along a projection vector, to a voxel with extreme density in the ultrasonographic volumetric data set. Multiple projection vector orientations are tested for image correlation, to accomodate any probable skew between the ultrasonographic and radiographic projections.

This application is a continuation-in-part of application Ser. No.09/322,129, filed on May 27, 1999, to which priority is claimed under 35U.S.C. 120.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates generally to ultrasonic and radiographicnon-invasive methods for examining tissue or other solids. Inparticular, the invention relates to the coordination or fusion ofultrasonic monograms with x-ray or other radiographic imaging techniquesto aid in the detection, location and biopsy of micro-calcifications ina human breast.

2. Description of the Related Art

Various means of non-invasive imaging are useful in medicine and otherfields for visually modeling the interior structure of a solid subjectbody. For example, a very common method of screening women for breastcancer is x-ray mammography. Ultrasonic imaging is another, less commontechnique for examining breast tissue.

X-ray mammography provides excellent detection of certain types oftissues, but nevertheless has shortcomings. This technique providesdetailed image information about well differentiated materials withinthe body (such as bone or other calcified tissue), but it performspoorly at discriminating between soft tissues with subtle differences indensity and structure. Some women have mammographically dense breasts,as compared to more fatty breasts; there is a substantially increasedrisk of missing breast cancers when diagnosing such women by x-ray. Theuse of x-rays for examination also necessarily results in the exposureof the patient to ionizing radiation, which has well know associatedrisks. The technique is also limited in that it projectsthree-dimensional structure onto a two-dimensional plane, and thus doesnot capture the elevation or depth (position in the direction ofradiation propagation) of features of interest

A newer imaging technique, ultrasonic imaging, is widely used fordiagnosis in numerous medical fields. When properly used and adjusted,an ultrasound imaging system can non-invasively provide across-sectional view of soft tissue being imaged, such as the tissue ofa breast, heart, kidney, liver, lung, eye, abdomen, or pregnant uterus.

A typical ultrasound imaging device operates by directing shortultrasonic pulses, typically having a frequency in the range of 1-30MHZ, into the tissue being examined. The device then detects responsessuch as echoes, harmonics, phase or frequency shifts, of the ultrasonicpulses caused by acoustic impedance discontinuities or reflectingsurfaces within the tissue.

A typical scanhead for an ultrasonic imaging system has a linear arrayof ultrasonic transducers which transmit ultrasonic pulses and detectreturned responses. The array of transducers provides simultaneous viewsof the tissue at positions roughly corresponding to the positions of thetransducers. The delay time between transmitting a pulse and receiving aresponse is indicative of the depth of the discontinuity or surfacewhich caused the response. The magnitude of the response is plottedagainst the position and depth (or time) information to produce across-sectional view of the tissue in a plane perpendicular to the faceof the scanhead.

Sophisticated ultrasonic imaging systems are available which are capableof volume reconstruction by assembling information from multipletwo-dimensional cross-sections to form a three dimensionalrepresentation of subject tissue. For example, one such system isdescribed in U.S. Pat. No. 5,787,889 to Edwards et al. (1998). Anenhanced ultrasound imaging system employing targeted ultrasound isdescribed in U.S. Pat. No. 5,776,062 to Nields (1998) Such systems arepotentially useful in the diagnosis of suspicious lesions in the breast.The system of Nields can also be used to guide the biopsy of a potentiallesion or suspicious mass in a breast. Compared to x-ray techniques,such ultrasonic techniques are advantageous in that the patient is notexposed to radiation. Ultrasound is also superior for imaging many typesof soft, low-density “hidden masses” which are typically invisible orvery obscure in x-ray imagery. On the other hand, the lower resolutionof ultrasonic imaging (compared to x-ray) makes it difficult orimpossible to identify fine features, such as hard micro-calcificationsin breast tissue, which would be visible in an x-ray.

Imaging of small calcifications is particularly useful because suchcalcifications play an important role in the detection of breast cancer.They are typically categorized as either benign, probably benign, orsuggestive of malignancy, based on a number of factors including size,shape, and distribution. Mammographically detected calcifications arefrequently the only detectable sign of breast cancer, so their properinvestigation is crucial. While some benign calcifications cannot bedistinguished from those associated with malignancy, many can be sodistinguished by their patterns and distribution. If more of thesebenign calcifications could be detected and characterized by carefulanalysis, the number of biopsies for benign conditions could bedecreased. Therefore, any imaging technique which can enhance theanalysis is extremely useful.

Although the smallest micro-calcifications are virtually impossible todetect by ultrasound, larger micro-calcifications, for example those ofaround 50 micron or greater size, do measurably affect ultrasoundpropagation (provided that short wavelengths are used). However, theirimages are not easily perceived in ultrasonographic imagery, because ofa characteristic of ultrasonographic imagery called “speckle” or“speckle noise”. Random or disorganized sound reflection andinterference cause ultrasonographic images to display a speckled orgrainy texture. A closely analogous phenomenon is commonly observed whencoherent light is used to view an irregular surface: the smooth surfaceappears grainy or speckled.

The speckle phenomenon tends to obscure the reading of ultrasonographsto detect micro-calcifications. The target micro-calcifications commonlyoccur as discrete, small individual members of a larger “constellation”or cluster (the shape of which depends on the type of calcification andits cause). By an unhappy coincidence, it so happens that the size ofthe individual micro-calcification members is often similar to thecharacteristic speckle size in many sonographs. In the case of multiplescattering sites per unit volume, the size of the characteristic speckleis a affected much more by the characteristics of the beamformingapparatus than the structure of the tissue being examined. Although aconstellation may be present, its recognition is made difficult by thespeckle noise in which the pattern is imbedded.

Another problem with breast imaging is the difficulty of combiningmultiple image modalities. Given that ultrasound and x-ray techniqueshave somewhat complementary imaging capabilities, it is often desirableto use both techniques to obtain the most imformation possible. Althougha patient (or other subject body) can be subjected to multiple imagingtechniques (for example x-ray and ultrasound), the images obtained arenot easily registered or correlated with one another. Differences inscale, position, or in the orientation of the plane of projection (of atwo-dimensional image) are almost inevitable.

U.S. Pat. No. 5,531,227 to Schneider (1996) discloses a method andapparatus for obtaining an image of an object obtained by one modalitysuch that the image corresponds to a line of view established by anothermodality. However, the method disclosed requires one or more fiducialmarkers to inter-reference the images. The preferred method disclosedalso involves mounting the patient's head immovably to a holder such asa stereotactic frame, which is inconvenient for the patient and thetechnicians. The method identifies fiducial markers by digitalsegmentation, feature extraction, and classification steps, which wouldmost suitably be performed with powerful digital hardware and customsoftware. The method disclosed will perform best with fiducial markerswhich are easily automatically recognized, as by some simple geometricproperty; it is described in connection with using circular eye orbitsas fiducial markers. In a human breast, however, such natural geometricfeatures may not be readily available.

Another method of correlating ultrasonic image data with radiographicimage data is disclosed in U.S. Pat. No. 5,640,956 to Getzinger et al(1997). This method requires that an x-ray image be obtained while thetissue is in the same position as it was while the ultrasonic data wasbeing gathered. It also requires the use of fiducial reference markers(preferably multiple x-ray opaque reference markers). The method andapparatus described by Shmulewitz in U.S. Pat. No. 5,664,573 similarlyrequires that the breast be maintained in the same position (relative tothe apparatus) during both the mammogram exposure and the ultrasoundimaging. Essentially, a mammographic system and an ultrasound imagingsystem are combined into a single, combined unit. Such a unit is bulkyand necessarily expensive.

U.S. Pat. No. 5,662,109 to Hutson describes a different approach tocorrelating and fusing ultrasonographic and mammographic data from thesame breast. In Huston's invention, an enhancer receives, combines, andcorrelates the two sets of data by a matrix manipulation technique. Theenhancer embeds the sets of data in matrices and uses singular valuedecomposition to compress the data into singular vectors and singularvalues. The compressed data can then be altered to enhance or suppressdesired features before display. This approach is promising but is verycomputationally demanding.

SUMMARY OF THE INVENTION

The invention is an apparatus and method for quickly coordinatingultrasonographic information about the internal structure of a solidsubject body with x-ray or other radiographic information taken from thesame subject body. The apparatus and method are especially suited fordetecting and enhancing visualization of micro-calcifications in humanbreast tissue.

Given a radiographic transmission image of a subject body, and givenfurther a set of volumetric, three-dimensional image data of the samesubject body, the invention relates a region in the originalradiographic image to a region within the three-dimensional image databy using a two-dimensional image cross-correlation, preferably performedby an optical correlator. In the preferred embodiment, the inventionalso uses a two-dimensional cross-correlation to find the elevation of afeature of interest in the three-dimensional data set.

In one embodiment, after coarse alignment of the images the inventionlocates, enhances, and displays small calcifications in a specifiedsub-region of the ultrasonographic image. Ultrasonographic imaging ofsmall calcifications or similar hard bodies is enhanced by correlatingan ultrasonographic data set with a radiographic image (preferablypre-processed and enhanced) of the same region of interest. A“constellation” or cluster of small calcifications is distinguished fromspeckle noise by the cross-correlation, which is quite sensitive to thecoincidence of a pattern of distributed small targets in both theultrasonographic and radiographic images, notwithstanding the presenceof random noise. An optical correlator is preferably used to perform therequisite high speed cross-correlations. Multiple projection vectororientations are tested for image correlation, to accomodate anyprobable skew between the ultrasonographic and radiographic projections,and a projection vector is found which acceptably aligns theradiographic and mammographic imagery at a fine level of allignment.After fine alignment of the mammogram and the sonogram is established inthe region of interest the three-dimensional position of an individualcalcification is preferably found by projecting from an identified pointin the radiographic image, along the projection vector, to a voxel withextreme density in the ultrasonographic volumetric data set. In this wayindividual calicifications in the three-dimensional ultrasonographicimage are located. An enhanced display is then generated and theinformation is optionally used to guide and/or verify a biopsy or otherdiagnostic follow up procedures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system level block diagram of an apparatus in accordancewith the invention;

FIG. 2 is a perspective view of one arrangement for obtaining x-rayimages for use by the invention;

FIG. 3 is a perspective view of one geometric setup which can be used toobtain ultrasonographic imagery for use by the invention;

FIG. 4a is a flow chart reading top-to-bottom and joining FIG. 4b at cutline A, showing initial steps of a procedure preferably used to find anoptimal transformation to correlate the ultrasonographic andradiographic images;

FIG. 4b is a flow chart continuing from FIG. 4a, joining FIG. 4a at cutline A and FIG. 4c at cut line B, showing the continuation of theprocedure of FIG. 4a;

FIG. 4c is a flow chart continuing from FIG. 4b and joining FIG. 4b atcut line B, showing the procedure used by one embodiment of theinvention to further relate the ultrasonographic and radiographic imagesin three dimensions;

FIGS. 5a, 5 b, and 5 c, show simplified examples of an input image, afilter template, and a resulting correlation output image, respectively,in an example of a cross-correlation operation which discoverspositional offset of correlated images;

FIG. 6 is a perspective view of an arrangement of planar slices of asubject body which can be selected for analyzing a subject body by theinvention;

FIG. 7 is a symbolic diagram of an optical correlator used in theinvention;

FIG. 8 is a block diagram of an apparatus in accordance with a variationof the invention suitable for imaging small calcifications in breasttissue;

FIG. 9a is a flow diagram reading top to bottom and joining FIG. 9b atcut line d, showing a method in accordance with the invention forimaging small calcifications in breast tissue;

FIG. 9b is a flow diagram continuing from FIG. 9a and joining FIG. 9a atcut line d, showing the continuation of the method;

FIG. 9c is a flow diagram continuing from FIG. 9b and joining FIG. 9b atcut line g, showing the further continuation of the method; and

FIG. 10 is a perspective view of an exemplary imaging geometry whichillustrates and defines skew angles which are variables used in themethod of FIGS. 9a and 9 b.

DETAILED DESCRIPTION OF THE INVENTION

In FIG. 1 a conventional ultrasound imaging system 10 scans a subjectbody 12 (for example, a human breast) to obtain three-dimensionalultrasonographic imagery revealing the composition of the subject body,by analysis of echos returned from internal structures of the subjectbody. This ultrasonographic imagery is provided in digital form to animage processor 14 for further analysis, processing and storage. Anx-ray or other radiographic device 16 examines the same subject body(breast), in general from an angle which may be different from the angleused by the ultrasonographic imaging system 10. The position androtation of the breast can and generally will differ from that usedduring the ultrasonography as well. The radiographic examination may bedone at approximately the same time as the ultrasonography, but this isnot essential. The x-ray or other radiographic imagery is digitized(either directly from detectors or indirectly using a digital imagescanner/digitizer 18 on a conventional x-ray photograph). Imageprocessor 14 processes and stores the digitized radiographic imagery,preferably in an x-ray image database. The image processor 14 thencontrols an optical correlator 20 to analyze correlations between theprocessed and stored radiographic and sonographic imagery.

The image processor 14, which could be either a specialized processor ora general purpose computer, is programmed to apply various coordinatetransformation to the radiographic and/or the ultrasonographic imagery.For example, the image processor 14 may apply various multiplications ofscales, rotations, and translations. The resulting processed imagery,ultrasonographic and radiographic, is then output to the opticalcorrelator 20, which returns a correlator output image to the imageprocessor 14. By exploiting the high speed and image processingcapabilities of presently available optical correlators, the imageprocessor 14 is able to quickly perform numerous test correlations onvariously transformed ultrasonographic and radiographic images,producing numerous correlator output images from optical correlator 20,with varying degrees of correlation. Although the ultrasonographic andradiographic images are not identical, common features visible in bothimages will produce a maximum degree of correlation when the images arealigned with one another and oriented in the same manner. The imageprocessor 14 selects the transformations which produce the optimaloutput correlation, then applies these transformations to at least oneof the radiographic and ultrasonographic images, thereby properlyaligning the images.

In the preferred embodiment of the invention, the image processor 14further correlates the transformed (properly scaled, translated, androtated) radiographic information with data subsets (planes or surfaces)of the three-dimensional ultrasonographic information to provide asophisticated diagnostic imagery which integrates both sources ofinformation and reveals the variation in the degree of correlation fromone plane to another. The original and the correlated images, asselected by a user via input device 22, are displayed on display(s) 24(for example, a video monitor and/or a printer) for viewing. Thecombined data may also be output from image processor 14 to a storagedevice 26 which stores the information and transformation parameters ona computer readable medium.

The invention thus coordinates the two independent sets of imaging data,thereby providing more complete imagery for analysis (typically by aphysician). The ultrasound imagery provides superior soft tissueinformation, while the radiographic imagery provides superiorinformation on calcified or hard masses; the invention coordinates thetwo information sets and, in the preferred embodiment, creates acorrelated or fused three-dimensional data representation of the subjectbody for display and analysis.

The detailed operation of the invention and its various subsystems arebest described with reference to an example of a particular coordinatesystem, to aid in visualization. FIG. 2 shows how the sonographic andradiographic images or information might be obtained from a human breastpresented for diagnosis, a typical application of the invention(although the techniques are not limited to use in medicine). The systemis conveniently illustrated in a Cartesian, rectilinear coordinatesystem having linear, perpendicular axes x, y and z, but the inventionis not limited to such a coordinate system.

Assume a human breast 32 is positioned between an upper and lowerpressure plates 33 and 34, as shown in FIG. 2. An x-ray or otherradiographic image is formed in the conventional way by illuminating thebreast with radiation 36 propagating downwards in the figure, with raysparallel to the z axis (toward -z). By exposing an x-ray sensitive plate38 in the conventional manner an image is formed which indicates theprojection of the breast's radiation absorption onto the plate 38 whichlies in the x-y plane. Alternatively, a radiation detector or detectorsin the x-y plane can be used to provide a two-dimensional array ofabsorption values by scanning the absorption at each point in the plane.Either method results in an image: a projection of the internalstructures of the breast onto a plane. The image is scanned anddigitized by scanner/digitizer 18 for storage and processing by imageprocessor 14.

Mammographic equipment is commercially available from a variety ofcommercial sources which will perform the above described functionssubstantially in the manner described.

Either before, after, or simultaneously with the radiography, the samebreast 32 is scanned with ultrasound by the ultrasonic imaging system10, preferably as shown in FIG. 3. With a patient preferably sittingfacing the imaging system, the patient's breast 32 is preferablyslightly compressed between upper and lower pressure plates 33 and 34 asin the x-ray setup. To produce the most consistent results, the breastshould preferably be under the identical pressure (between pressureplates 33 and 34) as was used in the previous radiographic technique. Inpositioning the breast between the pressure plates, edges of the plateswill contact the patient's chest above and below the breast. Because ofinconsistencies in positioning the patient's breast in the imagingsystem, the x′, y′ and z′ axes of FIG. 3 do not in the general caseexactly correspond with the x, y and z axes of FIG. 2, but may differ bya coordinate transformation: for example, they may differ by translationin the x or y directions, and by rotation about any axis. Rotation aboutthe z axis is especially likely.

With the breast 32 in position, conventional ultrasonic scanning ispreferably performed in planes (or surfaces), which will in general benon-parallel to that of the radiographic image (x-y plane in FIG. 2).FIG. 3 shows a typical geometry in which the scanhead 44 includes alinear array of ultrasonic transducers aligned parallel to the y′ axis.The scanhead 44 transmits ultrasonic pulses in the directions of theparallel lines 46, which are preferably perpendicular to the x′-y′ planeand parallel to the z′-y′ plane. The array of transducers in scanhead 44probe the underlying tissue lying (approximately) on lines 46 bydetecting returns of the ultrasonic pulses caused by acoustic impedancediscontinuities or reflecting surfaces within the tissue. The delay timebetween transmitting a pulse and receiving an return is indicative ofthe depth of the discontinuity or surface which caused the return. Acharacteristic such as magnitude, phase, or frequency of the returns isdigitized and is plotted against the depth (z′ axis) information and theinformation from the multiple transducers (dispersed in the y′direction) is assembled to construct an array representing across-sectional view of the tissue in a slice 48 parallel to the y′-z′plane and lying under scanhead 44.

Multiple slices can be scanned either by providing multiple scanheads, atwo-dimensional scanhead array, or by moving the scanhead across thebreast, for example in the y direction in FIG. 3. Only a few of themultiple slices, specifically slices 48, 50, and 52, are shown. Inpractice a large number of slices is desirable, for better resolution. Acomplete set of such slices is preferably scanned to form a threedimensional information set for at least some region of interest (ROI)chosen from the breast, which is preferably stored in a data structure(such as a three-dimensional array) to represent a three-dimensionalimage.

Ultrasonographic equipment is available commercially which can be usedas the ultrasonographic imaging system 10 described above.Three-dimensional information is desirable but not necessary to theinvention. A two-dimensional array of ultrasonographic data can becorrelated by the invention, but with a diminished amount of usefulinformation in the resulting display, as described below.

Once the ultrasonographic image has been scanned by ultrasonographicimaging system 10, the image processor 14 uses the optical correlator 20to determine the most correct scalings, rotations and translations ofthe two-dimensional radiographic information to properly relate it tothe ultrasound image derived from the same breast. This is necessarybecause the breast may not be (and in general will not be) oriented inprecisely the same position during the acquisition of the ultrasoundinformation as during the x-ray or radiographic exposure. The scale ofthe radiographic image relative to the ultrasonographic images may alsovary slightly, in spite of any attempts at cross-calibrating theradiographic and the ultrasonographic equipment.

FIG. 4a shows the preferred procedure by which the image processordetermines the proper coordinate transformations of scale, position, androtation which will coordinate the ultrasound data with the radiographicdata. The image processor 14 accesses (step 62) the stored ultrasounddata from ultrasonographic imaging system 10 and extracts (step 64) atwo-dimensional representation by projecting or “collapsing” thethree-dimensional ultrasound data onto a single plane. One method ofdoing this is by “cumulative projection”: a projection of thethree-dimensional data set onto a two-dimensional plane by summing thedata entries along vectors which are perpendicular to the projectionplane. One such vector, vector 66, is indicated on FIG. 3 forillustration. The density values associated with the voxels (threedimensional discrete volume cells) such as voxel 70 are summed along thevector 66. The summation of those density values yields a scalar valuewhich indicates the sum of the tissue densities along that vector. Thisscalar value is associated with the pixel 72 at the intersection of thevector 66 with the x′-y′ plane. Repeating this summation for multipleparallel vectors results in a set of values which defines the projectionof the three-dimensional sonographic imagery onto the x′-y′ plane.

Referring again to FIG. 4a, either before, after or during extractingthe two-dimensional ultrasonic data (for example, by the previouslydescribed method), the image processor 14 accesses (step 74) thetwo-dimensional x-ray information from an x-ray database and preferablypre-processes (step 76) the x-ray imagery. The x-ray pre-processing 76can include any of a variety of known image processing techniquesincluding (without limitation) contrast modification, smoothing,geometric transformation, thresholding or the selection of a region ofinterest. Depending on the type of optical correlator 20 used, asdiscussed in detail below, this step may also include two-dimensionalFourier transformation of the digitized x-ray image to prepare it forsubsequent optical correlation in a Vanderlugt optical correlator.Pre-processing may also (optionally) be applied (step 78) to theultrasound image data.

Next the image processor 14 adjusts the relative scale of the two imagesso that they at least approximately correspond in scale. This can bedone by various methods. For example, one method is to match the totalarea of the cross section of the breast area between the outer outlineand the chest wall in both images. In this method, the images shouldpreferably first be processed to remove low contrast features, leavingonly the easily visible outline of the breast and the chest wall. Thearea between these features in the two dimensional images is thenmeasured, for example by numerical integration by the image processor14. The area should correspond in both images, after taking into accountknown peculiarities of the imaging methods. For example, certainultrasonographic breast imaging systems fail to image the outer 10 to 20per cent of the breast. If the two areas do not correspond (aftercorrecting for peculiarities), a scaling factor, equal to the ratio ofthe areas, is calculated and applied to equalize the scales.

It is possible in many cases to maintain a previously known relativescale between the ultrasound and the x-ray apparatus. In such cases,little or no re-scaling is necessary.

Another simple method of determining relative scale is by placing afiducial marker or markers in both the x-ray and the ultrasonographicimage. For example, distinctive metallic foil or wire markers can beplaced a known distance apart, under the breast while both images areacquired. Alternatively, a single marker of known size could be used.The resulting distinctive mark or marks can be recognized in theacquired images, either by visual inspection or by processing by imageprocessor 14, to compare the imaged markers. It is then simple to findthe scaling factor by dividing the apparent separation (or size) of themarkers in one image by their apparent separation (or size) in the otherimage. The image processor then saves the scale factor and uses it totransform the images to a common scale. It is not necessary to maintainthe position of the marker(s) with respect to the breast to recover theproper scale factor.

Once the radiographic and ultrasonographic images have been transformedto a common scale, the image processor 14 determines the rotation andtranslation necessary to align the images, preferably by interactivelyperforming the steps grouped within the instruction loop 80 in FIG. 4a.First, two variables are initialized (step 88): a counter j to controlexecution of an instruction loop, and an associated rotation angleα_(j). Next, a cross-correlation of the dual images is computed (step90). Preferably this step is accomplished by using an optical correlator20 to perform the actual correlation computation, under control of theimage processor 14. (The details of the optical correlator are discussedbelow, in connection with FIGS. 5 and 6) The cross-correlation (step 90)produces a two-dimensional correlation output image indicating thedegree of cross-correlation, which is stored (step 92) along with theassociated rotation angle α_(j). The image processor then checks (step94) the counter variable to discover whether it has completed aprescribed number of iterations of the instruction loop 80.

Next, if the counter variable j has not reached jmax, the imageprocessor 14 continues and rotates (step 96) one of the dual imagesrelative to the other by some angular increment, for example by rotatingthe x-ray image one degree about an axis centered in the x-ray frame andparallel to the z axis. The counter is incremented (step 98) and theprocedure loops back to step 90 to perform another cross-correlation,this time with the images rotated incrementally. The procedure isrepeated until some number (jmax) of differently rotated correlationshas been performed. The parameter jmax should be chosen to be largeenough so that the range of the associated angle α_(j) encompasses theanticipated maximum possible range of rotation. For breast examinationin the geometry shown in FIG. 3, a range of 45 degrees is adequate.

The rotations applied in step 96 are not limited to rotations about asingle axis, but could include rotations about multiple independent axes(or, equivalently rotations about an axis obliquely oriented withrespect to the orthogonal axes shown). This allows the correction for anoblique viewing angle of one image with respect to the viewing angle ofthe other.

After the counter j reaches jmax the image processor 14 exits theinstruction loop 80. The procedure continues as diagramed in FIG. 4b.The correlation output images previously stored in the variousiterations of step 92 are compared (step 100) with one another to findthe correlation output image with the maximum correlation, and itsassociated angle of rotation α_(j). The value α_(m) (the rotation anglewhich produced maximum correlation) is stored (step 102) and either theultrasonographic imagery or the radiographic image is rotated (step 104)by α_(m) to bring it into the same orientation as its counterpart.Preferably the radiographic image is rotated to align with the(three-dimensional) ultrasonographic imagery, because rotating thetypically larger database associated with the three-dimensionalultrasonographic imagery would require more computations than arerequired to rotate the two-dimensional radiographic database.

It should be understood that in addition to discovering the correctscale and rotation angles, the cross-correlation (step 90) in the abovedescribed procedure produces an output image which reveals the correcttranslation (position shift) which best aligns the dual images. Thetranslation information is contained in the result of thecross-correlation operation (preferably in two-dimensions) defined as:

H(x,y)=g(x,y)*ƒ(x,y)=∫∫ƒ(α,β)g(x−α,y−β)dαdβ  (1)

Where f and g are functions of two variables (images), x and y are thespatial variables of the two-dimensional images, α, and β are dummyvariables of integration, H is the cross-correlation of functions f andg, and the range of integration is across the entire image. If f and gdiffer only by a positional offset in x and y, H(x,y) will have a sharppeak at a position x_(p),y_(p), which is displaced from a central,correlation alignment position (typically defined as x=0, y=0) by adisplacement corresponding to the offset between f and g. This wellknown result has been used to recognize and locate features of initiallyunknown locations in a field by reference to a template. See, forexample, Russ, John C., The Image Processing Handbook (CRC Press, 1992),pp. 218-24. A simplified example is shown in FIGS. 5a-5 c. The inputimage, a star 106 at an offset position shown in FIG. 5a, is correlatedwith a filter derived from the centered star-shaped template 107 shownin FIG. 5b. The resulting correlation output shown in FIG. 5c has a peak108 at a position x_(p),y_(p), corresponding to the offset between theinput image and the filter template. To align the images with acorrelation peak at position x_(p),y_(p) it is sufficient merely totranslate one of the images by a displacement equal to the offsetx_(p),y_(p).

If the subject breast 32 is differently oriented in the x-ray and theultrasonographic image, it will most likely be rotated slightly about avertical axis (parallel to z axis) which lies somewhere within thepatient's body. This is equivalent to a translation plus a rotation ofthe image frame about its center, both of which are discovered andcorrected by the procedure of FIGS. 4a through 4 c.

Returning to FIG. 4b, the image processor 14 next analyzes (step 114)the correlation output image to locate the positional offset of thecorrelation peak from an aligned correlation position, then translates(step 116) one image relative to the other as necessary to better alignthe images. After finding the optimal combination of rotations, scalingsand translations which properly align the radiographic with theultrasonographic information, the image processor 14 preferably stores(step 118) the transformed images and the transformation parameters inits associated memory and preferably outputs (step 120) the transformedimages to a display device 24. The visual output can be displayed invarious forms, and should preferably be structured to make apparent thespatial relationship between the x-ray and the ultrasonographic images.Various display formats can be used to allow the simultaneous display ofboth data sets in proper relation to the common coordinate systemdiscovered by the invention. For example, overlays, color coding,projection onto various planes, topographic quasi-three-dimensionaldisplay formats could be used, in various combinations. The transformedimages and the transformation parameters are also preferably stored bystorage device 26 on a computer readable medium.

In one embodiment the invention provides additional three-dimensionalinformation about the subject body by further correlating theultrasonographic and the radiographic images in order to locate thedepth (z coordinate) of such features. To locate the depth of featuresof interest, the ultrasonographic imagery is first partitioned by imageprocessor 14 into conveniently defined slices, for example slice 124 asshown in FIG. 6. Each slice includes one or more layers of the threedimensional ultrasonographic image data. The slices are defined andcalculated by image processor 14, for example by summing data pointsalong vertical vectors such as 125, to collapse multiple thin layersinto a thicker slice (a “partial cumulative projection”). In FIG. 6, forexample, three thin layers 126 of ultrasonographic imagery (shown onlypartially to clarify FIG. 6) might lie between bottom slice 124 and theimmediately overlying slice 127. The partial cumulative projection istaken point-by-point, by summing the ultrasonographic image values atpoints along vectors such as vector 125, and accumulating the result tothe point 128 where vector 125 intersects the slice 124. The accumulateddata values at each defined point on slice 124 collectively define theslice.

In a typical application, slices of 5 millimeters in thickness aresuitable. Although planar slices are convenient for purposes ofillustration, in some applications the slices might usefully be takenalong non-planar contours. Such slices are also within the scope of theinvention. Thinner slices are desirable for better depth definition withthin features, but may result in poor correlation if the feature ofinterest is distributed in depth by more than the defined slicethickness. The x′, y′,z′ axes of FIG. 6 are not, in general, identicalto the x,y,z axes of FIG. 2; rather, they may be rotated (about anyaxis) and translated (by an x,y,z vector) with respect to the x,y,zaxes.

FIG. 4c shows one procedure which is suitably used (preferablysubsequent to the procedure of FIGS. 4a and 4 b) in an embodiment of theinvention which locates the depth of solid features of interest. Suchfeatures are preferably user selected (step 129) by input to the imageprocessor 14 from a user input device 22. In this embodiment, havingpreviously determined and applied the proper scale, rotation andtranslations to align the x-ray information and stored the transformedimages, the image processor 14 defines slices (step 130) as discussedabove in connection with FIG. 6. The image processor 14 then initializes(step 132) a counter variable n. This variable is used to index multipleparallel planar slices which intersect the z′, axis at differentpositions.

The image processor 14 then selects (step 134) a planar slice at depthZ_(n) from the three dimensional ultrasound data, for example slice 125which is parallel to the underlying transformed x-ray image plane (thex′-y′ plane in FIG. 6). The extracted slice is then correlated (step136) with the transformed x-ray image or with a user selected ROI takenfrom the x-ray image. The image processor 14 controls correlation of theslice (which is preferably performed by the optical correlator 20),stores (step 138) the correlation array and its associated depth Z_(n)in memory , checks (step 140) the counter to find out whether thecounter has reached maximum n, and if not, increments (step 142) acounter, then loops back to select another slice at a different depthfor correlation (step 134).

After a programmed number (maximum n) of slices have been selected andprocessed, the counter n reaches maximum value nmax, corresponding tothe uppermost plane 135 in FIG. 6. The plane with the maximumcorrelation for the feature of interest and its associated depth Z_(n)are stored in memory (step 150). One or multiple features of interestmay be selected and correlated, and the resulting plane(s) of maximumcorrelation used to construct (step 152) a multi-planar,three-dimensional data structure identifying the three-dimensionallocations of the features of interest.

The three-dimensional imagery constructed by the procedure of FIG. 4c ispreferably then displayed visually (step 154) to a user by the display24. As described above, various means of symbolic representation orthree-dimensional projection graphics can be used to present theinformation effectively. The three-dimensional construction is alsopreferably stored by storage device 26 in a computer readable medium.

By determining the depths of features of interest, the invention can beused to compose or construct a composite, three dimensionalrepresentation of the subject body, incorporating information from bothultrasound and x-ray imagery. Thin features (with low thickness in the zdirection) will result in a particularly good correlation which is wellrestricted to the slice at the actual depth of the feature, with poorcorrelation in other slices.

Combining the ultrasound and the x-ray images by the invention is initself useful for displaying or analyzing internal structures of a solidbody. In addition the invention offers specific advantages in analyzingor diagnosing biological tissues. For example, to diagnose human breastcancer it is helpful to detect and locate prognostic indicators(tell-tale signs of cancer which lie at or near the cancerous lesion).Some prognostic indicators, such as hard micro-calcifications, are mosteasily seen in x-ray images. Others, such as a density of atypicallobules, are much more easily seen in ultrasound derived images. Bycombining the two images into a single data structure which properlyrelates the x-ray and ultrasound images in space, a physician canvisually determine whether two such prognostic indicators are occurringat the same location in the breast. Since the randomly causedcoincidence of two such statistical indicators is improbable (providedthat the two indicators are truly independent), the spatial coincidenceof two independent in-situs could provide a stronger indication of amalignant lesion than either independent observation would provide. Bycoordinating the x-ray and the ultrasound imagery, the invention thusprovides a new diagnostic tool.

In the procedures depicted in FIGS. 4a, 4 b and 4 c, it is highlypreferable that the correlation operations be carried out by an opticalcorrelator. In the preferred embodiment, the image processor 14electronically writes the dual images to the optical correlator 20. Theoptical correlator 20 preferably performs the correlation operations andreturns a resulting correlation image to the image processor 14.

Optical correlators use wave optics to correlate images in twodimensions by first performing essentially a two-dimensional spatialFourier transform on a two-dimensional source image. This method takesadvantage of a well known mathematical property of the Fouriertransform: many operations including correlation are more easilycalculated in the Fourier transform domain than in the original spatialdomain. Specifically, a two-dimensional correlation operation is definedby equation 1 (above), where f(x,y) and g(x,y) are the two-dimensionalfunctions or images to be cross-correlated, and α and β are dummyvariables of integration. This operation can be performed digitally foreach point x,y by numerical techniques, but a very large number ofcalculations are required even for one image correlation. Performingsuch an operation digitally is very time consuming and requiresinconvenient lengths of time on any but the fastest digital computers.

Unlike a conventional digital computer, an optical correlator can veryrapidly perform a correlation operation, correlating a source image witha filter image by (1) optically Fourier transforming a source image, (2)comparing the source and filter image in the Fourier transform domain,and then (3) performing an inverse Fourier transformation to produce thecorrelation pattern in a spatial representation. An optical correlatorcan accomplish these operations much faster that a digital computerbecause the optical Fourier transformation is executed as a simultaneousoperation on all points of the source image, using inherent propertiesof wave optics to generate the Fourier transform in two dimensions. Thespeed of the device is limited for practical purposes only by theavailable read and write speed of the data transfer to the correlator;the actual optical processes occur in fractions of a nanosecond intypical optical correlators.

The principles of the optical correlator are known, and have beendescribed for example in the U.S. Pat. No. 5,311,359, to Lucas et al.Compact optical correlators suitable for use in the present inventionare commercially available from Litton Data Systems, Inc., in AgouraHills, California, as well as from other sources. Alternate types ofoptical correlators such as the Joint Transform Correlators described inU.S. Pat. No. 5,650,855 to Kirsch et al., U.S. Pat. No. 5,216,541 toTaksue et al. or U.S. Pat. No. 5,438,632 to Horner, may also be employedwith the invention.

For purposes of describing the present invention, the optical correlator20 may be considered functionally as an electro-optical device havingthree (electronic) ports, as shown in FIG. 7. The three ports include:(1) an image input port 160 for receiving an electronic signal encodingan input image for correlation; (2) a filter input port 162 forreceiving a second electronic signal encoding a second image or “filter”for correlation; and (3) an output port 164, typically from a chargecoupled device (CCD) imager, which converts the correlation image intoelectrical form for output. In addition the device requires a source(not shown) of preferably coherent electromagnetic radiation, typicallya laser, which provides the medium used for computation.

Both the image input port 160 and the filter input port 162 are realizedas two-dimensional spatial light modulators (SLMs) organized astwo-dimensional image matrices, with addressable image pixels (typicallyarranged in the familiar row and column pattern). Accordingly, the inputimage must be formatted (suitably by image processor 14) to fit thematrix; and each pixel of data should preferably be addressed, undercontrol of the image processor 14, to the spatially corresponding pixelon the SLM. For example, in one embodiment of the invention, the imageinput port and the filter input port are realized as 256×256 pixilatedmatrices. Accordingly, in this embodiment the image processor 14, aspart of pre-processing steps 64 and 76 (in FIG. 4a), maps theultrasonographic and the x-ray data onto a 256×256 matrix for output tothe optical correlator 20. In a typical embodiment of the invention aVanderlugt type optical correlator is used. In such a correlator the“filter” image must be pre-processed by two-dimensional Fouriertransformation. In such an embodiment the image written to the filterport is preferably Fourier transformed by image processor 14 (forexample in pre-processing step 76), to provide a frequency domainpattern. In an alternate embodiment, a joint transform correlator may beused as optical correlator 20. This eliminates the need for the digitalFourier transformation of the filter image, as the transformation isoptically performed by the joint transform correlator.

When the input the filter images have been written to the input andfilter ports 160 and 162, the optical correlator produces an outputimage which is a two dimensional output correlation pattern having anoptical peak or peaks (bright spot) at the position of greatestcorrelation between the collapsed sonographic image and the radiographicimage. The degree of correlation is indicated by the intensity of theoutput signal. The position of the output peak on the two-dimensionalmatrix of the correlator output CCD indicates the translations or shiftsof the images relative to one another. The output image is read from theoutput photodetector (CCD) 164 by the image processor 14 in theconventional manner, typically by shifting the CCD voltage values outsequentially in rows (or columns) and then digitizing the output levels.

Although the invention is described in terms of linear transformationsof the coordinates, such as translation, rotation, and scalemultiplication, the invention is not limited to linear transformations.Non-linear transformations of coordinate systems may be useful in someapplications. For example, the ultrasonographic information may beobtained with the breast differently deformed, as by a change ofposition of the subject, or by instrument pressure. By applying amathematical transformation, which may in general be non-linear, abetter mapping of the deformed subject breast onto the original subjectbreast can be obtained. Similarly, some scanning techniques may involvecurvilinear, non-cartesian coordinate systems which would be treatedwith non-linear transformations.

Imaging Microcalcifications in Breast Tissue:

One specific variation of the invention is particularly adapted forimaging breast micro-calcifications and for targeting breast biopsy ofsuch micro-calcifications. In this variation the invention is preferablyfirst used as previously described to coarsely register a radiographicand an ultrasonic image of the subject breast; the invention then moreclosely examines a smaller region of interest (ROI)in the ultrasonographand mammograph, to locate micro-calcifications at a finer level ofresolution. (Alternatively, coarse registration could be initiallyprovided by other means.)

FIG. 8 shows a system which can be used in accordance with the inventionto locate breast micro-calcifications in ultrasonographicthree-dimensional imagery. The system is similar to that of FIG. 1. Afilm scanner/digitizer 18 and/or a digital data network 210 providedigital radiographic imagery to the image processor 14. At least one offilm scanner/digitizer 18 or data network 210 is required, or anequivalent image input channel for radiographic imagery. The imageprocessor 14 is preferably a 64 bit workstation such as the SiliconGraphics 02, although less powerful processors could be used at theexpense of speed or resolution. The image processor 14 is also linkedwith a breast ultrasonography system 10 for obtaining ultrasonographicbreast imagery. Preferably, the image processor 14 is also linked withan Optical correlator 20 (as previously discussed, for processing andcomparing ultrasonographic and radiographic imagery). A user inputdevice 22 and display 24 are also linked to the image processor 14, toallow control/command input and image display to the user. Optionally, abiopsy apparatus 212 is also provided with access to the subject breast12, most preferably linked to the image processor 14. When this optionis present, information about the position of the biopsy instrument canbe fed back visually to the user via feedback path 214 (shown as abroken line) from the subject breast 12, via ultrasonographic apparatus10 through the image processor 14 to display 24.

To recognize micro-calcifications in an ultrasonographic image, it isdesirable to identify a region of interest in the imagery which includesa “constellation” or cluster of micro-calcifications. As used herein,“constellation” means a grouping or pattern of multiple smallcalcifications. The word “constellation” is chosen because it suggests aspatially related pattern of distributed calcifications, which can berecognized by its spatial arrangement, much in the way that astronomicalconstellations are recognized by the spatial relationships amongindividual stars. Moreover, as will be discussed below, the inventionrecognizes such “constellations” of micro-calcifications by theirspatial pattern in a two dimensional projection; analogously,astronomers easily recognize stellar constellations as projectedtwo-dimensional patterns even though the actual member-stars aredispersed throughout a three-dimensional volume.

FIG. 9a illustrates the flow of a method suitable for identifying andlocating micro-calcifications in ultrasonographic imagery, in accordancewith the invention. The various steps are preferably executed underprogram control by the image processor 14, responsive to user input fromthe input device 22 (in FIG. 8). First, in an embodiment having morethan one data entry channel (both a digital data network 210 and a filmscanner 18, for example), the desired data entry channel is selected(step 230), based on the patient imagery which is available. Forexample, if film is available which provides the best resolution of theROI, the film scanner 18 would suitably be selected in response tooperator input. Next, the film is scanned (step 232 a) or the digitaldata is requested from the LAN (step 232 b).

Next, the ultrasonographic and the digitized radiographic images areapproximately registered by a “coarse registration” step 234. Eitherconventional registration methods can be used, or, preferably, theimages can be registered by the method described above in connectionwith FIGS. 4a through 4 c, by correlating macroscopic features visiblein both sets of imagery. The result is an approximate degree ofregistration of the dual images, sufficient to locate a region ofinterest in the mammogram and a corresponding region of the breast underultrasonographic examination.

After coarse registration step 234, a region of interest (ROI) in theimagery is defined (step 236) in both the mammogram and the subjectbreast, either by operator input or by reference to manually appliedmarkings on the radiograph. For example, it is common for a physician tocircle or otherwise mark a region of interest on a mammogram, and themarkings can be used to define the region of interest. Alternatively, adigital mammogram can be cropped digitally according to operator input,or the ROI can be identified by a computer aided diagnosis system. Theregistration previously determined in step 234 then allowsidentification of the corresponding region in the subject breast 12under ultrasonographic examination.

After identification, the ROI in the mammogram is preferably rendered ata higher resolution (step 238), either by re-scanning film or byrequesting a higher resolution partial image from a digital datanetwork. Scanning a smaller region of interest at higher resolutionallows finer inspection without impractical expansion of the requisitedata set. The corresponding ROI in the actual breast is then scanned bythe ultrasonographic equipment 10, also at a high resolution setting(step 240). Preferably very short wavelength ultrasound, in the regionabove seven Megahertz, should be used.

Referring now to FIG. 9b, from the high resolution ultrasound athree-dimensional image data set is then developed and stored forprocessing (step 242). The host processor next processes (step 244) theultrasonographic three-dimensional high resolution data and enhances thecontrast of micro-calcifications. Preferably the processor usesphase-shift detection to detect the micro-calcifications as accuratelyas possible. Even with enhancement, some false targets and noise(speckles) will be present, complicating the detection process. Based onthe enhanced three dimensional image, a set of derived images is thencreated by projecting the high resolution data along an assumedprojection vector P (step 246) Preferably, a maximum value projection iscreated, in which the pixel in the plane of projection has an intensityvalue equal to the maximum intensity of any voxel which lies on thevector of projection, P. Alternative or additional techniques ofprojection could be used: for example, a sum of display intensities ofvoxels along a projection vector could be used, with similar results.The projective geometry is discussed further below, in connection withFIG. 10.

Next, the processed radiographic image and a version of the derivedultrasonographic projection image are two-dimensionally correlated (step248), preferably by using the optical correlator 20 (shown in FIG. 8)under control of the image processor 14 to calculate the correlationbetween the images. Specifically, if a Vanderlugt-type correlator isused, a subject image is written to an electronically addressable inputSLM, while the calculated two-dimensional Fourier transform of acomparison image is written to a “Filter” SLM as previously described inconnection with FIG. 7. A resulting correlation image is formed and readfrom a photodetector array and postprocessed by the image processor 14to determine a degree of correlation between the images. In particular,the optical correlator serves the image processor 14 to calculate atwo-dimensional image which approximates the cross-correlation functionH_(x,y) defined by:

H _(x,y) =Σ/iΣ/j F _(x+i,y+j) ·G _(i,j)

where F and G are the input image functions which vary over (discrete)indices x and y, and i, j, are dummy variables of summation. Note thatthis cross-correlation is the discrete approximation to the continuouscorrelation function previously given (equation 1, above). The functionH can also be normalized with respect to average image intensity bydividing by a normalizing factor, for example, the quantity:

{square root over (Σ_(iΣ) _(j) F _(x+i,y+j) ² ·Σ/iΣ/j G _(i,j) ²+L )}

For a discussion of the correlation operation, and the application ofthe Fourier transform to its calculation, see Bracewell, Ronald N., TheFourier Transform and its Applications, (McGraw-Hill, second edition).

Alternatively, as previously discussed, the cross-correlation functioncould be digitally calculated either in spatial or Fourier transformdomain, but optical processing will greatly increase the speed ofprocessing.

The image processor 14 preferably computes at this stage (step 248) aquality parameter Q which represents the intensity of the correlationbetween the derived ultrasonographic projection and the radiographichigh-resolution image of the ROI. A suitable parameter can be defined,for example, as the maximum intensity correlation at any pixel of thephotodetector, divided by the average intensity across the frame (tonormalize the signal). In general, noise will prevent perfectcorrelation. In addition, some micro-calcifications may be obscured orfor other reasons not detectable in at least one of the images. Mostprobably, perfect correlation will not be obtained because of slightvariations in the alignment of the ultrasound axes relative to the raydirection of the radiograph (a “skewed orientation”) The problem ofskewed orientation is geometrically illustrated in FIG. 10. A solid body260 is shown in relation to conventional axes X, Y, and Z and decomposedinto cubic voxels 262. This is an example of an arrangement which couldtypically be used to represent three-dimensional ultrasonographic data,by associating a data value with each voxel 262. To derive atwo-dimensional (projection) image, we require a direction ofprojection. We assume projection along vectors parallel to a generalizedvector of projection P. As shown, P is not necessarily parallel to anyof the axes. We arbitrarily define a vector V, parallel to Z andintersecting the top surface of the volume 262 at its center. We canthen represent the direction of P relative to V by two Euler angles eand f as shown. The angle e is the angle between P and V in a planewhich contains P and V (the PV plane). The angle f defines the anglebetween the PV plane and an arbitrary plane which includes the V axis.For convenience, an arbitrary plane parallel to the ZX plane is chosen,indicated by the x′ axis. If a normal projection were chosen, P would beparallel to Z and both angles e and f would be zero. However, P is ingeneral not parallel to the axis Z. In a real imaging situation, smallangles e and f will inevitably exist. As a result, a constellation ofdiscrete calcifications dispersed within the volume 260 will beprojected onto the YX plane (a convenient plane for image projection)with some slight distortion (relative to a normal projection).Fortunately, the distortion can readily be calculated by well knowtrigonometric relationships. For simplicity, it can be suitably assumedthat all radiographic (x-ray) vectors in the initial image acquisitionare essentially parallel. Although divergent ray, pinhole projection orsimilar x-ray techniques are known and widely used, the general approachof the invention is to obtain sufficient resolution from essentiallyparallel ray radiographs and to correlate these with ultrasonography toprovide greater definition.

Returning to FIG. 9b, initially in step 246 it is assumed that theprojection vector P (defined by the angle of the mammographic rays) isparallel to the Z axis (defined by the axis of ultrasonographic dataorganization). As previously described, a derived image is developed andcorrelated with the mammographic image, resulting in a correlation imageand a value Q (steps 246 and 248). Preferably, to obtain best resolutionthe derived images should be reiteratively recalculated using anincrementally increasing series of assumed valued for the Euler angles eand f. For example, the assumed angle (relative to Z) could be sweptoutward from Z in a spiralling search pattern. Alternatively, concentriccircle patterns other any other pattern could be used. It should not benecessary to search wider that ten degrees. Thus the method tests forcompletion of the sweep range (decision box 270). If the range has notbeen exhausted, the process changes the angles e and/or f, saves Q (step272) and loops back (step 274) to the prior step 246 to deriveultrasound images; the correlation then repeats the in step 248 with animage derived according to an incrementally varied projection angle.After a complete range of angles is swept, the angles e and f whichproduced the maximum quality parameter Q are saved, the ultrasoundimagery is preferably rotated to the corresponding optimal angles e, fto align with the radiographic imagery, and the quality parameter issaved (all step 276).

Note that the rotation of the three-dimensional ultrasonographic dataset (in steps 272 and 276) could be accomplished either by (a) digitallymanipulating the data set to transform to a rotated coordinate system,or (b) changing the scan angle by physically moving the ultrasound scanhead or the subject breast in relation to the scan head. Either methodor a combination are contemplated within the intended scope of theinvention.

Preferably the maximum quality parameter Q should be tested in step 276for an acceptable level of correlation, which may be arbitrarily orsubjectively predetermined by operator input. If at an acceptable levelof Q is obtained, a potential target detected in the ultrasonographicthree-dimensional ROI is identified as a true micro-calcification if theindividual target lies on a vector which would project, (aftercorrection for skew) onto a corresponding detected micro-calcificationimaged in the radiographic image (step 278). The depth along theultrasound vector which is associated with each true target (as measuredby time delay) is also recorded and saved by the image processor 14(step 280), thereby locating the target. After this step is performedpreferably for each and all targets of interest, The image processor 14uses the resulting data to construct a three-dimensional display (step282) which shows the “true” micro-calcifications (those detected andcorrelated from both images to a pre-defined degree of correlation). Theimage is then displayed for a user and optionally recorded for futurereference.

The volumetric display which results can be usefully coded with colorcoded voxels, a topographic representation, or with simulated surfacevolumetric display techniques which are known. See for example thetechniques discussed in John C. Russ, The Image Processing Handbook (CRCPress, 1992), pages 402-426. The display can then be used to guidefurther diagnostic workup.

In one variation of the invention, the three-dimensional model can beused to guide biopsy procedures (step 284). This is done by extractingthe targeting coordinates of operator selected calcifications, thenpassing the coordinates (which are correctly given with respect to thecurrent orientation of the ultrasonographic equipment) to the biopsyapparatus. Biopsy instruments are typically easily imaged by theultrasonographic equipment and can be guided to the target coordinatesto sample one or more specific, targeted calcifications. Thus, theenhanced three-dimensional information and/or display are preferablyused to guide a biopsy of one or more operator selected calcifications,for example by guiding a fine needle aspiration instrument or a vacuumassisted core biopsy instrument to the target calcification (step 284,optional). Any biopsy instrument which is discernable by ultrasonographycould suitably be used. Post-biopsy imaging (ultrasonographic orradiographic) is preferably also used as follow up to verify that thecalcifications were correctly sampled, in accordance with appropriatemedical standards of care.

While illustrative embodiments of the invention are described above, itwill be evident to one skilled in the art that numerous variations,modifications and additional embodiments may be made without departingfrom the invention. For example, the construction of the ultrasoundimaging system, the geometries and coordinate systems employed, or thetype of radiographic image (including the radiographic medium) may bevaried. The invention may be applied to multiple or stereotacticradiographic images to correlate them to an independent source ofultrasonographic imagery. Computer aided tomographic (CAT) images, oreven magnetic resonance images (MRI) may be correlated withultra-sonographic imagery. Various means of data storage, transmission,or processing may be employed, even to the extent that the imaging(X-ray and/or ultrasound) may be performed at one location and time,then transmitted via data network to another location for processing inaccordance with the method of the invention at a later time. Theresolution or type of image that is sent from the image processor to theoptical correlator could also be altered. Three-dimensionalcross-correlations are also possible (but computationally complex). Tothe extent that such operations can be decomposed into multiple planaroperations, the use of the optical correlator as described above couldgreatly accelerate computations. Accordingly, it is intended that theinvention be limited only in terms of the appended claims.

We claim:
 1. A method of imaging small objects such as calcifications ina bodily tissue, comprising the steps of: providing radiographic imagedata from an identified region of interest (ROI) of the tissue using afirst imaging unit; providing three-dimensional ultrasonographic imagedata corresponding to substantially the same ROI either before or aftersaid radiographic image data is provided and using a second imaging unitindependent of said first imaging unit such that said ROI need not beidentically positioned for both radiographic and ultrasonographic imageacquisition; and generating an image of the objects based upon acoincidence of data in the radiographic and the ultrasonographicimagery, said coincidence determined by cross-correlation of (a) theradiographic image data, and (b) a two-dimensional reduction of thethree-dimensional ultrasonographic image data.
 2. The method of claim 1,wherein said two-dimensional reduction comprises multiple pixels, and adata value at a pixel is computed by taking an extremum data value fromdata associated with positions along a projection vector whichintersects said pixel.
 3. The method of claim 1, wherein saidtwo-dimensional reduction comprises multiple pixels, and a data value ata pixel is computed by taking a sum of data associated with positionsalong a projection vector which intersects said pixel.
 4. The method ofclaim 1, further comprising the step of: finding the position of anobject by projecting back along the projection vector from a targetimage in the radiographic image to an extremum data value along saidvector in the ultrasonographic data.
 5. The method of claim 1, furthercomprising: displaying a visual representation of the ROI on a display;and enhancing the visibility of a target object within the display basedupon the congruence of (a) an assumed projection of the target objectand (b) a radiographic indication of a dense object.
 6. The method ofclaim 5, wherein the assumed projection of the target object is found bysearching for a geometric relationship between the radiographic data andthe ultrasonographic data which cross-correlation value in apredetermined range.
 7. A method of imaging small objects such ascalcifications in a bodily tissue, comprising the steps of: providingradiographic image data from an identified region of interest (ROI) ofthe tissue; providing three-dimensional ultrasonographic image datacorresponding to substantially the same ROI; and generating an image ofthe objects based upon a coincidence of data in the radiographic and theultrasonographic imagery, said coincidence determined bycross-correlation of (a) the radiographic image data, and (b) atwo-dimensional reduction of the three-dimensional ultrasonographicimage data, wherein the cross-correlation is performed by an opticalcorrelator.
 8. A method of imaging small objects such as calcificationsin a bodily tissue, comprising the steps of: providing radiographicimage data from an identified region of interest (ROI) of the tissue;providing three-dimensional ultrasonographic image data corresponding tosubstantially the same ROI; and generating an image of the objects basedupon a coincidence of data in the radiographic and the ultrasonographicimagery, said coincidence determined by cross-correlation of (a) theradiographic image data, and (b) a two-dimensional reduction of thethree-dimensional ultrasonographic image data, wherein thecross-correlation comprises computing an image wherein the image valueat each position x,y approximates the cross-correlation function H_(y)defined by: H _(x,y) =Σ _(i)Σ_(j) E _(x+i,y+j) ·G _(i,j)  where F and Gare image functions to be cross correlated, x and y are positionsindices of the images, i and j are dummy indices of summation and thesummation is across the image in two dimensions.
 9. A method of locatingone or more small, radiographically dense objects within a subject body,comprising the steps of: providing a radiographic image of an area ofinterest (ROI) including the object using a first imaging unit; derivingan image from three-dimensional ultrasonographic volumetric data eitherbefore or after said radiographic image data is provided and using asecond imaging unit independent of said first imaging unit such thatsaid ROI need not be identically positioned for both radiographic andultrasonographic image acquisition; cross-correlating the derived imagewith the radiographic image; identifying the target object based on apredetermined degree of correlation between the images; and displaying avisual interpretation derived from the ultrasonographic volume data set,in which a probable target object has its visibility enhanced based on acoincidence of image data in the radiographic and ultrasonographicimages.
 10. The method of claim 9, wherein the step of providing aradiographic image comprises: scanning a portion of a radiographic filmwith a digitizing scanner, and storing the resulting data asradiographic image data.
 11. A method of locating one or more small,radiographically dense objects within a subject body, comprising thesteps of: providing a radiographic image of an area of interest (ROI)including the object; deriving an image from three-dimensionalultrasonographic volumetric data; cross-correlating the derived imagewith the radiographic image using an optical correlator; identifying thetarget object based on a predetermined degree of correlation between theimages; and displaying a visual interpretation derived from theultrasonographic volume data set, in which a probable target object hasits visibility enhanced based on a coincidence of image data in theradiographic and ultrasonographic images.
 12. A method of locating oneor more small, radiographically dense objects within a subject body,comprising the steps of: providing a radiographic image of an area ofinterest (ROI) including the object; deriving an image fromthree-dimensional ultrasonographic volumetric data; cross-correlatingthe derived image with the radiographic image, wherein thecross-correlating step comprises computing an image wherein the imagevalue at each position x,y approximates the function H, defined by: H_(x,y) =Σ _(i)Σ_(j) F _(x+i,y+j) ·G _(i,j)  where F and G are imagefunctions to be cross correlated, x and y are positions indices of theimages i, j are dummy indices of summation and the summation is acrossthe image in two dimensions; identifying the target object based on apredetermined degree of correlation between the images; and displaying avisual interpretation derived from the ultrasonographic volume data set,in which a probable target object has its visibility enhanced based on acoincidence of image data in the radiographic and ultrasonographicimages.
 13. A method for guiding a biopsy instrument to sample acalcification or other small sample from a human body, comprising thesteps of: providing radiographic image data regarding of a region ofinterest (ROI) from the human body using a first imaging unit; providingultrasonographic image data from substantially the same ROI eitherbefore or after said radiographic image data is provided and using ascanning ultrasonographic apparatus independent of said first imagingunit such that said ROI need not be identically positioned for bothradiographic and ultrasonographic image acquisition; locating acalcification in relation to the ultrasonographic apparatus, bycross-correlating said radiographic and said ultrasonographic image datato register the image data; and guiding the biopsy instrument to thecalcification based upon its determined location.
 14. A method forguiding a biopsy instrument to sample a calcification or other smallsample from a human body, comprising the steps of: providingradiographic image data regarding of a region of interest from the humanbody; providing ultrasonographic image data from substantially the sameregion, by scanning ultrasonographic apparatus; locating a calcificationin relation to the ultrasonographic apparatus, by cross-correlating saidradiographic and said ultrasonographic image data to register the imagedata, said cross-correlating step performed by an optical correlator;and guiding the biopsy instrument to the calcification based upon itsdetermined location.
 15. A method for guiding a biopsy instrument tosample a calcification or other small sample from a human body,comprising the steps of: providing radiographic image data regarding ofa region of interest from the human body; providing ultrasonographicimage data from substantially the same region, by scanningultrasonographic apparatus; locating a calcification in relation to theultrasonographic apparatus, by cross-correlating said radiographic andsaid ultrasonographic image data to register the image data, said crosscorrelation step comprising computing an image wherein the image valueat each position x,y approximates the function H, defined by: H _(x,y)=Σ _(i)Σ_(j) F _(x+i,y+j) ·G _(i,j)  where F and G are image functionsto be cross correlated, x and y are positions indices of the images, iand j are dummy indices of summation and the summation is across theimage in two dimensions; and guiding the biopsy instrument to thecalcification based upon its determined location.
 16. A method forguiding a biopsy instrument to sample a calcification or other smallsample from a human body, comprising the steps of: providingradiographic image data regarding of a region of interest from the humanbody; providing ultrasonographic image data from substantially the sameregion, by scanning ultrasonographic apparatus; locating a calcificationin relation to the ultrasonographic apparatus, by cross-correlating saidradio-graphic and said ultrasonographic image data to register the imagedata, said correlation calculated in a frequency domain representation;and guiding the biopsy instrument to the calcification based upon itsdetermined location.
 17. A system for enhancing imagery of bodilytissues by relating ultrasonographic and radiographic images,comprising: an image processor, programmed to: (a) receive saidultrasonographic and said radiographic images, (b) process said imagesto derive processed ultrasonographic and radiographic images, and (c)control an optical correlator to compute cross-correlation images fromsaid processed images; and an optical correlator coupled to said imageprocessor and arranged to correlate said processed images and to outputto said image processor a cross correlation image which is indicative ofthe correlation between the processed images.
 18. The system of claim17, wherein the image processor is programmed to relate either theradiographic image or the ultrasonographic image to a spatial coordinatesystem, and to transform at least a portion of the other of the imagesto the spatial coordinate system.
 19. The system of claim 17, whereinthe image processor is further programmed to locate at least one ofmultiple small targets distributed in the bodily tissue, by calculatinga three-dimensional configuration of said targets consistent with boththe ultrasonographic and the radiographic image data.
 20. The system ofclaim 17, further comprising an ultrasonographic imaging system,arranged to communicate ultrasonographic image data to said imageprocessor, for imaging a bodily tissue ultrasonographically.
 21. Thesystem of claim 17, further comprising a high resolution film scanner,arranged to scan and digitize radiographic film of a subject bodilytissue and to communicate data to said image processor for analysis. 22.The system of claim 17, wherein said image processor communicates with adata network to receive image data regarding a subject bodily tissue.23. The system of claim 17, wherein said image processor communicateswith a biopsy apparatus, to aid in guiding a biopsy instrument toward atarget.
 24. The system of claim 17, further comprising: a biopsyapparatus, in communication with said image processor to aid in guidinga biopsy instrument toward a target.